Computer system for forensic analysis using motion video

ABSTRACT

A computer system calibrates an image from digital motion video, which originated from a camera that has a view of a scene from a period of time, with an image rendered from a three-dimensional model of the scene for a view based on a location (position and orientation) of the camera in the scene. The calibrated image from the digital motion video can be overlaid on the rendered image in a graphical user interface. The graphical user interface can allow a user to modify opacity of the overlay of the calibrated image from the digital motion video on the rendered image. The overlaid image can be used as a guide by the user to provide inputs with respect to the three-dimensional model.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a non-provisional application of U.S. ProvisionalPatent Application Ser. No. 62/774,808, filed Dec. 3, 2018, entitled“Computer System for Forensic Analysis Using Motion Video”, which ishereby incorporated by reference.

BACKGROUND

Motion video is a common source of evidence of incidents such as trafficaccidents and criminal activity. A variety of information can be gleanedfrom motion video through image processing, such as position, speed,acceleration, and relative position of objects. Each image in motionvideo only provides two spatial dimensions of information. Motion videoalso provides a temporal dimension, allowing additional information tobe extracted by analyzing multiple images. However, the information thatcan be extracted is limited by what was actually visible to the cameraor cameras.

Another source of evidence can be a computer-based three-dimensionalmodel of a scene. Such a three-dimensional model, when presented usingthree-dimensional modeling, rendering, and animation software on acomputer, allows a user to visualize different areas of a scene, measuredistances, and determine relative positions of objects, and so on.However, in the context of forensic analysis of incidents such astraffic accidents and criminal activity, such three-dimensional modelsgenerally do not include evidence about pertinent objects that werepresent during an incident of interest. For example, a witness at ascene of a vehicle accident may recall something visible from theirperspective at the time of the accident. Because any three-dimensionalmodel of the scene of the accident would be prepared at a time otherthan the time of the accident, the witness would not be present in thatmodel. To illustrate a view from the location of the witness poseschallenges in pinpointing the witness location in the scene. To providereliable evidence based on the witness' location in thethree-dimensional model, based on an approximation of what the witnesscould see, involves defining a margin of error.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is intended neither to identify key oressential features, nor to limit the scope, of the claimed subjectmatter.

A computer system calibrates an image from digital motion video, whichoriginated from a camera that has a view of a scene from a period oftime, with an image rendered from a three-dimensional model of the scenefor a view based on a location (position and orientation) of the camerain the scene. The calibrated image from the digital motion video can beoverlaid on the rendered image in a graphical user interface. Thegraphical user interface can allow a user to modify opacity of theoverlay of the calibrated image from the digital motion video on therendered image. The overlaid image can be used as a guide by the user toprovide inputs with respect to the three-dimensional model.

Further manipulations of the three-dimensional model, such as selectingpoints, from which measurements can be made or for adding andpositioning models of objects, can be guided by the information providedby the overlay. For example, a vehicle may be present in the calibratedimage but not in the three-dimensional model. Given the overlaid,calibrated image, a user can select points, through an input device,based on the visual guidance provided by the overlaid image. The inputsfrom the input device are passed, however, to the application supportingthe three-dimensional model. This application uses these inputs as ifthe overlaid image is not present, in effect allowing the user to selecta location in the three-dimensional model using the overlaid image as aguide. The inputs passed to the application can be used for any purposethe three-dimensional modeling or animation application supports.

While one of the applications described herein is forensic analysis,these techniques are applicable to any other application in whichinformation from an image or video may be useful to guide inputs to athree-dimensional model. Such applications include but are not limitedto historical reconstructions, real estate, interior design,architectural design, game design, animation, motion-pictures, and otherapplications of three-dimensional modeling and animation. An overlaidimage can be paired with any other application to allow a user to usethe image as a guide for providing inputs to the underlying application.

In the following description, reference is made to the accompanyingdrawings which form a part hereof, and in which are shown, by way ofillustration, specific example implementations. Other implementationsmay be made without departing from the scope of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a data flow diagram of an illustrative example implementationof a computer system incorporating three-dimensional models and motionvideo from a camera.

FIG. 2 is a flowchart of operation of an example implementation of acomputer system such as in FIG. 1.

FIG. 3 is an illustration of an example graphical user interface forremoving distortion of an image.

FIG. 4 is an illustration of an example graphical user interface forcalibrating an image from motion video from a camera to an image of arendered three-dimensional model.

FIG. 5 is an illustration of an example graphical user interfaceoverlaying an image from motion video from a camera on athree-dimensional model.

FIG. 6 is a flowchart of operation of an example implementation of acomputer system such as in FIG. 1 for adding a three-dimensional modelof an object to the three-dimensional model of a scene.

FIG. 7 is a block diagram of an example computer.

FIGS. 8A-8E are example images of using a calibrated image as a guide toplace a three-dimensional object in a three-dimensional model.

FIG. 9 is an example image of a margin of error report illustratingmeasurements made within a three-dimensional model using a calibratedimage as a guide.

DETAILED DESCRIPTION

As used herein a digital still image is made using a camera, in whichlight is captured by a light-sensitive device, such as a charge-coupleddevice (CCD), to generate a two-dimensional image on the device. Thecamera may include a lens to focus the light onto the light-sensitivedevice. This image in turn is output by the device and digitally storedas an N by M array (where N and M are positive integers) of pictureelements, called pixels. The image can be called a frame. Each pixel isrepresented by data representing one or more color components. “Digitalmotion video” is a time-based series of frames, with each frame storedas a digital still image. A digital still image can be stored in anuncompressed form or in a compressed form.

A “three-dimensional model” comprises digital data on a computerrepresenting, in three spatial dimensions, a plurality of objects inthose three spatial dimensions. Such an object can be defined by aplurality of vertices and edges between those vertices, or by dataspecifying one or more surfaces in three dimensions, or by sample datarepresenting points in three dimensions, or any other techniquespecifying an object in three dimensions. Such an object also may have aspecified position and orientation in three spatial dimensions. Anobject may be associated with texture data applied to a surface. Ananimation of a three-dimensional model includes any data that specifieshow the data representing an object may change over time.

FIG. 1 is a data flow diagram of an example implementation of a computersystem 100 that combines digital motion video from a camera with athree-dimensional model of a scene within the view of the camera.

Such a computer system 100 includes a source 102 of data representing athree-dimensional model 104 of a scene, and a source 106 of digitalmotion video 108 which originated from a camera that has a view of thescene and which is from a period of time. The three-dimensional model104 of the scene includes the location of the camera. Data specifyingthe location (position and orientation) 110 of the camera in thethree-dimensional model also is determined. The three-dimensional model104 and the digital motion video 108 can be stored as data files oncomputer storage.

The source 102 of the data representing the three-dimensional model canbe any of a variety of tools that provide such data. For example,three-dimensional laser scanning technology, currently available fromFARO Technologies, Inc., of Lake Mary, Fla., and Leica Geosystems, Inc.,of Norcross, Ga., can be used to capture highly accuratethree-dimensional measurements of a scene, from which athree-dimensional model can be generated. As another example, images canbe captured using a camera, such as a camera mounted to a mobile robotor autonomous flying vehicle (drone), from which a point cloud in threedimensions is generated. The Pix4D family of software products fromPix4D, S.A., of Switzerland can be used for this purpose, for example.As another example, an architectural or engineering model generated bycomputer-aided design software, such as the AUTOCAD product fromAutodesk, Inc, can be used to provide a three-dimensional model of ascene or of an object. As another example, an object model and animationgenerated by computer-aided modeling and animation software, such as theMAYA product from Autodesk, Inc, can be used to provide athree-dimensional model of a scene or of an object. The invention is notlimited to a particular kind of software for generating athree-dimensional model of the scene.

In practice, for forensic analysis, the entity performing the forensicanalysis may use a tool to acquire or generate the three-dimensionalmodel and may use computer software that can process thethree-dimensional model to render images based on the three-dimensionalmodel.

The source 106 of digital motion video can be any data storage systemthat stores digital motion video originating from a camera that has aview of a scene. The digital motion video can be provided in any of avariety of color formats (i.e., bits per pixel and color componentsused), spatial resolutions (i.e., image size in n pixels by m pixels),and temporal resolutions (i.e., frame rate or number of frames persecond). The term “color format” is intended to include both black andwhite images and color images. The digital motion video can includeimages from different cameras which are multiplexed into a single streamof images. The digital motion video can have associated audio. Thedigital motion video can originate from motion video captured in ananalog format, which is then converted to digital format. While thedescription refers to motion video, these techniques also can be appliedto any still image of a location.

In practice, for forensic analysis, the camera at a scene, and digitalmotion video originating and stored from it, are controlled by a thirdparty who provides access to the digital motion video to the entityperforming the forensic analysis. Generally, the digital motion video isprovided as a data file on computer readable storage. The entityperforming forensic analysis also may have another party create aninitial three-dimensional model of a scene. Notably, the digital motionvideo from the camera at the scene and the three-dimensional model ofthe scene thus can be created at different times and under the controlof different entities. The sources 106 and 102 of this information canbe “cloud” storage or other storage accessible over a wide area or localarea computer network. The digital motion video and three-dimensionalmodel can be uploaded to such storage by entities that created them.

In most cases, the lens of the camera may introduce distortion in theimage. In some implementations of this computer system, the computersystem can include an image processing module 120 that removes lensdistortion. Other types of distortions may exist, and the imageprocessing module 120 also may be programmed to remove these other typesof distortion. For example, the camera may be inside a housing and thematerial of the housing may introduce distortion. For example, manycameras for security systems have a housing with transparent materialbetween the camera lens and its surroundings. This transparent materialmay introduce distortion. The image processing module 120 has an inputthat receives one or more images from the digital motion video andprovides corrected digital motion video 122 as an output. An exampleimplementation of lens distortion correction will be described in moredetail below.

The corrected digital motion video may no longer be a rectangular imagedue to the correction. A cropping operation can be performed on thecorrected digital motion video, or a selected image from it, to make oneor more corrected images rectangular. The cropping operation can beautomatically or manually performed. Such cropping can be performedafter the corrected digital motion video is calibrated to an imagerendered using the three-dimensional model.

The corrected digital motion video 122, or a selected image from it, iscalibrated with the three-dimensional model 104 based on the cameralocation (position and orientation) 110 using a calibration module 130.The calibration module 130 uses an image 142 generated by rendering thethree-dimensional model 104 from the viewpoint of the camera location110. The calibration module uses correspondence points (not shown, butdescribed in more detail below in connection with FIG. 4), to align thedigital motion video 122 with the rendered image 142 of thethree-dimensional model. The user starts by specifying a viewpoint,having a position, orientation, and depth of view, in thethree-dimensional model that corresponds to the camera location 110. Inone implementation, the calibration results in a two-dimensionaltransform 180 applied to digital motion video to conform it to theviewpoint. When the two-dimensional transform is applied to an imagefrom the corrected digital motion video, the result is an image that ismatched in perspective, orientation, or alignment, with the view fromthe three-dimensional model.

Generally, for forensic applications, a single image from a clip ofdigital motion video is selected from the clip and used to performcalibration. A user-selected image or an arbitrarily selected image canbe used, but the computer system can be programmed to simply select thefirst image of the clip. After calibration parameters are established,each image from the clip can be processed to calibrate each image. Thecalibration parameters can be applied to other clips originating fromthe same camera for clips recorded under similar conditions.

The output resulting from application of the calibration module 130 is aspatially aligned pair of a calibrated image 132 from the correcteddigital motion video 122 with the rendered image 142 from rendering thethree-dimensional model 104 from the viewpoint of the camera location110. The system can output the 2D transform 180 and the correcteddigital motion video 122. Given the transform 180, corrected video 122,camera location (position and orientation) 110 and three-dimensionalmodel 104, any other computer program implementing such functionalitycan generate and display a calibrated image 132 and rendered image 142for display.

Given the calibrated digital motion video and the rendered image of thethree-dimensional model of the scene, a calibrated image 152 from thedigital motion video can be overlaid on the rendered image 142 in agraphical user interface. The graphical user interface can allow a userto modify opacity of the overlay of the digital motion video on therendered image. In the example implementation shown in FIG. 1, thecomputer system includes a rendering module 140 which receives thethree-dimensional model 104 and the camera location 110 as an input andrenders an image 142 of the three-dimensional model in a first displayarea (not shown in FIG. 1). A playback module 150 receives thecalibrated digital motion video and displays one or more images 152 in asecond display area (not shown in FIG. 1). The playback module can haveassociated playback controls to play, pause, scan forward and backward,and stop display of the calibrated digital motion video in the seconddisplay area. See FIG. 5 below for an example of overlaid images andassociated controls.

Given the displayed image 152 from the calibrated digital motion videoand the rendered image 142 of the three-dimensional model, inputs froman overlay control module 160 allow a graphical user interface 170 todisplay the calibrated image 152 overlaid on the rendered image 142 witha specified amount of opacity. The overlay control module, in responseto user inputs to the overlay control module, sets an opacity for theoverlay of the calibrated image 152 over the rendered image 142. Thegraphical user interface can receive user inputs to move the image 152within the graphical user interface to position it over the renderedimage 142 of the three-dimensional model. While these overlaid imagesare displayed, the displayed image 152 may be updated to a differentcalibrated image from the digital motion video based on user inputs tothe playback module. Also, other user inputs can be provided to therendering module 140, or the three-dimensional modeling or animationsoftware incorporating it, to perform additional operations within thatsoftware.

There are several ways in which the overlay control module 160 can beimplemented.

In some implementations, the rendering module 140 is part ofthree-dimensional modeling or animation software, and the playbackmodule 150 is a separate computer program that displays an image in itsown display area. The overlay control module 160 can be a program thatprocesses inputs from the user and directs the user inputs to theappropriate computer program, such as a. to the rendering module 140 orthe software of which the rendering module is a component, or b. to theplayback module 150, or c. to the graphical user interface, or d. toitself to use the user inputs to change the overlay controls such asopacity.

In some implementations, the overlay control module 160 can include anyone or more of the playback module 150, the calibration module 130, orthe image processing module 120.

In some implementations, the rendering module and the playback modulemay be part of the three-dimensional modeling or animation software, andthe overlay control module 160 may generate graphical user interfaceelements that allow a user to manipulate display controls, such asopacity of the displayed image 152, within the rendering module 140 ofthe three-dimensional modeling software.

In some implementations, the rendering module and the playback modulemay be part of digital motion video playback software, and the overlaycontrol module may generate graphical user interface elements that allowa user to manipulate display controls, such as opacity of the displayedimage 152, within the digital motion video playback software.

In some implementations the overlay control module can have two states:a first state in which inputs received from the user are processed bythe overlay control module, and a second state in which inputs receivedfrom the user associated with points in the displayed calibrated imageare passed through to the three-dimensional modeling or animationsoftware. The overlay control module may have a setting allowing a userto toggle between these two states.

In an example, illustrative implementation, on a computer with a Windowsoperating system, the overlay control module can be implemented as afirst application having two windows, one providing controls and anotherproviding the overlaid calibrated image (See FIG. 5). The window for theoverlaid calibrated image can be implemented by setting an extendedwindow style for the window, using the SetWindowLong* methods of theWindows operating system, to enable transparency of that window (thestyle currently called “WS_EX_TRANSPARENT”). This extended window styleallows the window to be considered transparent and let click events passthrough to windows underneath it. Using this implementation, such anoverlay control module can be paired with any other application to allowa user to use the image as a guide for providing inputs to theunderlying application.

While the calibrated images from the digital motion video, as overlaidon the rendered three-dimensional model, are displayed, a variety ofuseful operations can be performed by users, particularly for advancingforensic analysis of an incident that was captured in the digital motionvideo.

For example, the images in the digital motion video may have capturedobjects that are not present in the three-dimensional model. Using theimage from the digital motion video as a guide, locations related tothose objects can be identified within the reference of thethree-dimensional model. Distance, location, size and other informationcan be extracted based on the identified corresponding points in thethree-dimensional model. By allowing the user to adjust opacity whileviewing the overlaid images, the user can more accurately selectlocations in overlaid images. By passing through the selected pointsfrom the graphical user interface directly to the three-dimensionalmodelling or animation software, locations within the 3D model can beselected. With such a configuration, the use of calibrated images fromdigital motion video to enhance selection of points in athree-dimensional model can be applied to any 3D modeling or animationprogram without modifying that program.

As another example, using the image from the digital motion video as aguide, a three-dimensional object, corresponding to an object in thedigital motion video, can be added to the three-dimensional model. Theuser can adjust the size, position, and orientation of the object in thethree-dimensional model. By allowing the user to adjust opacity whileviewing the overlaid images, the user can more accurately orient theobject in the three-dimensional model. By adding the object to thethree-dimensional model, further analysis of that object, such asadditional measurements, animation, viewpoints, and the like, can beperformed.

Turning now to FIG. 2, an example flow-of-operation using the computersystem of FIG. 1 will now be described. The computer system receives(200) digital motion video. This digital motion video generally isreceived as a data file over a computer network into computer readablestorage, or on a form of computer readable storage. Typically,independently, the computer system also receives (202) athree-dimensional model of a scene related to the digital motion video.This model generally is received as a data file over a computer networkinto computer readable storage, or on a form of computer readablestorage.

Given the digital motion video, the computer system the corrects (204)any distortion in the digital motion video, such as distortion caused bythe lens of the camera. Any other image processing also may beperformed.

The computer system calibrates (206) the corrected digital motion videowith the three-dimensional model, based on a location of the camera usedto capture the digital motion video within the space represented by thethree-dimensional model.

After calibration, the computer system generates (208) the userinterface with an image from the digital motion video overlaid on arendered image from the three-dimensional model, along with controlsallowing the position and opacity of the overlay to be controlled. Insome implementations, a user may position a display area for the digitalmotion video over a display area for the rendered three-dimensionalmodel and align them manually. In some implementations, the computer maydisplay them in alignment. The displayed overlay controls allow a userto adjust the opacity of the displayed calibrated image from the digitalmotion video. In some implementations, the overlay controls include anability to lock the display areas in position with respect to eachother. In some implementations, the overlay controls include an abilityto toggle the state of an overlay control module to pass throughlocation inputs to the underlying three-dimensional modeling oranimation system.

Given the overlaid display area, the computer system can process (210)further user inputs, such as those that adjust opacity of the overlay orthat provide inputs to either the rendering of the three-dimensionalmodel or the display of the digital motion video. For example, locationsin the three-dimensional model can be selected, and different operationson and manipulations of the three-dimensional model can be selected andperformed based on the input locations. For example, different imagesfrom the digital motion video can be displayed.

Having now described the general operation of the computer system withrespect to the example implementations of FIGS. 1 and 2, some examplegraphical user interfaces will now be described in connection with FIGS.3 through 5.

FIG. 3 is an illustration of an example graphical user interface forremoving distortion of an image.

FIG. 3 illustrates a graphical user interface in which an original image300 from the digital motion video is displayed. A user specifies inputpoints 302 indicating one or more objects in the image that shouldrepresent a straight line. It is preferable to instruct the user toprovide input points closer to the edge of the image where there is moredistortion. Because the same distortion is applied to each image by thelens, the same correction can be applied to each image. In someimplementations, the computer system computes a mathematicaltransformation to apply to the image such that the specified points forma straight line in a resulting image. Generally, such a transformationis a two-dimensional spatial transform of an image. The resulting imageshown at 304 is not rectangular. The user interface can be provided witha control (not shown) allowing the user to crop the image.

As one example for correcting lens distortion, the computer prompts auser to enter plurality of points along a line that appears curved inthe image, but which the user knows should be straight. The computerthen computes lines between the first point and the second point alongthe curve, then from the second point to the third point, then from thethird point to the fourth point and so forth. The computer thencalculates the difference between the angles of each line. Then, twovalues, labeled K1 and K2, typically called distortion coefficients, forthe image are iteratively adjusted over a known range and the angles ofthe lines between the points is recalculated for each K1 and K2 value.The algorithm selects the final K1 and K2 for the lens distortion valuesbased on the minimum possible difference between the angle of each line.K2 can be set to K1/−10. Example implementations for correction suchdistortion are found at least in: the ImageMagick library of imageprocessing software from ImageMagick Studio LLC, also described in“Correcting Lens Distortions in Digital Photographs”, by WolfgangHugemann, 2010, available athttps://www.imagemagick.org/Usage/lens/correcting_lens_distortions.pdr,hereby incorporated by reference; “Correction of Geometric PerceptualDistortions in Pictures”, by Denis Zorin and Alan Barr, in Proc. ACMSiggraph 1995, pp. 257-264, hereby incorporated by reference; or “LensDistortion Correction using a Checkerboard Pattern”, by Seok-Han Lee,Jae-Young Lee, and Jong-Soo Choi, in Proceedings of The 7th ACM SIGGRAPHInternational Conference on Virtual-Reality Continuum and ItsApplications in Industry (VRCAI '08), Article No. 44, 2008, herebyincorporated by reference.

FIG. 4 is an illustration of an example graphical user interface forcalibrating a three-dimensional model to an image from motion video froma camera.

After the digital motion video is processed to account for distortions,the processed image (400) is presented next to a corresponding renderedimage (402) of the three-dimensional model of the scene fromapproximately the same viewpoint as the camera that provided the digitalmotion video. The computer system prompts the user to entercorrespondence points (examples of which are shown at 404-411) toindicate points in the images that correspond. In some implementations,the correspondence points may be determined automatically by thecomputer system.

Given this information (the correspondence points), the computer systemcan calibrate the two images. Generally, at least four points are used,but typically more than four points are used. A two-dimensionaltransform is computed to transform an image from the digital motionvideo to match the viewpoint specified in the three-dimensional model.The transform is specified using conventional techniques based on thespecified viewpoint and the correspondence points. As an exampleimplementation, using the Open Source Computer Vision Library (OpenCV),a “findHomography” method can be used to determine a transform based onthe correspondence points. Given the transform, applying the transformis a geometric operation applied to the corrected image from the camera.As an example implementation, OpenCV has a “perspectiveTransform” methodthat can be used.

A least-squares approach can be used when interpolating pixels based onweighted color values when applying the transform. As an exampleimplementation, a “rigid transformation” can be used, as described in“Image Deformation Using Moving Least Squares”, by Schaefer, S.,McPhail, T., and Warren, J., in Proceedings of ACM SIGGRAPH 2006, pages533-540, hereby incorporated by reference.

FIG. 5 is an illustration of an example graphical user interfaceoverlaying an image from motion video from a camera on athree-dimensional model.

In FIG. 5, an image from the digital motion video is displayed at 500and overlaid on an image of the rendered three-dimensional model, asindicated at 502. A control panel can be provided at 504 to allow a userto input an opacity parameter, as indicated at 506. In this example theopacity control is shown as a slider. Any other way of inputting a valuefor the opacity, which generally is a value representing a percentagebetween 0 percent and 100 percent, or a value between 0 and 1, can beused. An input and control indicating whether the positions of thedigital motion video 500 and rendered image 502 are “locked” also can beprovided as indicated at 508.

In one implementation, the overlay controls and overlay image canoriginate from one application and can be two different “windows” of thesame application. The window with the overlay image can be placed overthe window of a three-dimensional modeling software which displays arendered image. After positioning and sizing the overlay image over thewindow with the rendered image, a user can instruct the computer to“lock” those positions and maintain the windows in the same relativeposition.

Turning now to FIG. 6, an example flow-of-operation for using thecomputer system of FIGS. 1 through 5 will now be described. This exampleis for use of the digital motion video as a guide to insert athree-dimensional model of an object, from the scene as captured in thedigital motion video, into the three-dimensional model of the scene.

In FIG. 6, the computer system receives (600) a three-dimensional modelof an object, such as a model of a vehicle. In response to user input,the three-dimensional model of the object is placed (602) by thecomputer system in the three-dimensional model of the scene. In someimplementations, the user may manually adjust the position, orientation,and size of the object, using the image of the object from the overlaiddigital motion video as a guide. In some implementations, the computersystem may automatically or semi-automatically position, orient, andscale the object to match the image of the object in the overlaiddigital motion video.

Given the placement of the three-dimensional model of the object intothe three-dimensional model of the scene, further manipulations andmeasurements can be performed using the three-dimensional model. Forexample, the computer system can provide (604) measurements of size anddistance with respect to the additional object. As another example, thecomputer system can provide (606) other renderings of thethree-dimensional model with the object, such as from differentviewpoints. For example, if the object is a vehicle, then the computersystem can render a view of the three-dimensional model from theviewpoint of an operator of the vehicle.

FIGS. 8A-8E are example images of using a calibrated image as a guide toplace a three-dimensional object in a three-dimensional model. The firstimage in FIG. 8A is an example frame from video which was captured at ascene and which has already been calibrated and overlaid on top of athree-dimensional model of the scene. The second image in FIG. 8Billustrates only the three-dimensional model background. In FIG. 8B, thebus that is visible is a three-dimensional object that was placed by theuser in the model, using the image of the bus from the video so that thebus in the model is aligned with the bus from the video. Thus, the userplaces the model of the bus back into the scene within thethree-dimensional modelling software. The next two images FIGS. 8C and8D depict the same scene and three-dimensional model, but for a newposition of the bus based on the position of the bus from one framelater in the video sequence. The position of the bus in each frame canbe represented as a form of animation of the bus in a three-dimensionalanimation of the model. The final image in FIG. 8E is a screenshotshowing the same scene from a new viewpoint in the three-dimensionalmodel to allow a different view of the position of the bus than providedby the video.

In some implementations, a measurement can be provided with dataindicating an estimated accuracy of the measurement. For example,instead of an exact value, a measurement can be represented by a valueand a range around that value. The range around a value can bedetermined based on known sources of error in measurements. For example,in any measurement, there may be an error due to sampling used to createthe three-dimensional model, and due to the pixel resolution of thedigital motion video, and due to transformations applied to the digitalmotion video. As an example, in some areas of an image, a pixel of animage may represent one inch of actual distance. Thus, any measurementof an object in that area of the image is accurate only within one inch.Given the known sources of errors, these errors can be accumulated toprovide a measure of accuracy associated with any spatial measurement.Some example techniques for computing such measurements are described,for example, in “The reverse projection technique in forensicphotogrammetry”, by J. Whitnall and F. H. Moffitt, in Non-TopographicPhotogrammetry, 2nd ed. (H. Karara, ed.), American Society forPhotogrammetry and Remote Sensing, Falls Church, Va., pp. 389-393, 1989,hereby incorporated by reference; or in J. Russ and B. Neal, 2016, TheImage Processing Handbook, 7th edition, CRC Press, 13:978-1-4987-4028-9,hereby incorporated by reference.

In some implementations, the computer system can output images and data,for example in the form of a report, indicating, within an image, anobject of interest, corresponding measurements, and corresponding dataabout the margin of error for such measurements. As an example, inresponse to a user selecting two points through an overlaid image, theunderlying three-dimensional model may provide coordinates in threedimensions, or a measurement between those coordinates. Informationabout the scanner accuracy, calibration accuracy, and resolutionaccuracy can be combined to provide a margin of error for themeasurements related to the object of interest. FIG. 9 is an exampleimage of a margin of error report illustrating measurements made withina three-dimensional model using a calibrated image as a guide.

Having now described an example implementation, FIG. 7 illustrates anexample of a computer with which components of the computer system ofthe foregoing description can be implemented. This is only one exampleof a computer and is not intended to suggest any limitation as to thescope of use or functionality of such a computer. The various modulesand graphical user interfaces in FIGS. 1 through 6 can be implemented inone or more computer programs executed on one or more such computers asshown in FIG. 7.

The computer can be any of a variety of general purpose or specialpurpose computing hardware configurations. Some examples of types ofcomputers that can be used include, but are not limited to, personalcomputers, game consoles, set top boxes, hand-held or laptop devices(for example, media players, notebook computers, tablet computers,cellular phones including but not limited to “smart” phones, personaldata assistants, voice recorders), server computers, multiprocessorsystems, microprocessor-based systems, programmable consumerelectronics, networked personal computers, minicomputers, mainframecomputers, and distributed computing environments that include any ofthe above types of computers or devices, and the like.

With reference to FIG. 7, a computer 700 includes a processing systemcomprising at least one processing unit 702 and at least one memory 704.The processing unit 702 can include multiple processing devices; thememory 704 can include multiple memory devices. A processing unit 702comprises a processor which is logic circuitry which responds to andprocesses instructions to provide the functions of the computer. Aprocessing device can include one or more processing cores (not shown)that are multiple processors within the same logic circuitry that canoperate independently of each other. Generally, one of the processingunits in the computer is designated as a primary processor, typicallycalled the central processing unit (CPU). A computer can includecoprocessors that perform specialized functions such as a graphicalprocessing unit (GPU).

The memory 704 may include volatile computer storage devices (such as adynamic or static random-access memory device), and non-volatilecomputer storage devices (such as a read-only memory or flash memory) orsome combination of the two. A nonvolatile computer storage device is acomputer storage device whose contents are not lost when power isremoved. Other computer storage devices, such as dedicated memory orregisters, also can be present in the one or more processors. Thecomputer 700 can include additional computer storage devices (whetherremovable or non-removable) such as, but not limited to,magnetically-recorded or optically-recorded disks or tape. Suchadditional computer storage devices are illustrated in FIG. 7 byremovable storage device 708 and non-removable storage device 710. Suchcomputer storage devices 708 and 710 typically are nonvolatile storagedevices. The various components in FIG. 7 are generally interconnectedby an interconnection mechanism, such as one or more buses 730.

A computer storage device is any device in which data can be stored inand retrieved from addressable physical storage locations by thecomputer by changing state of the device at the addressable physicalstorage location. A computer storage device thus can be a volatile ornonvolatile memory, or a removable or non-removable storage device.Memory 704, removable storage 708 and non-removable storage 710 are allexamples of computer storage devices. Computer storage devices andcommunication media are distinct categories, and both are distinct fromsignals propagating over communication media.

Computer 700 may also include communications connection(s) 712 thatallow the computer to communicate with other devices over acommunication medium. Communication media typically transmit computerprogram instructions, data structures, program modules or other dataover a wired or wireless substance by propagating a signal over thesubstance. By way of example, and not limitation, communication mediaincludes wired media, such as metal or other electrically conductivewire that propagates electrical signals or optical fibers that propagateoptical signals, and wireless media, such as any non-wired communicationmedia that allows propagation of signals, such as acoustic,electromagnetic, electrical, optical, infrared, radio frequency andother signals.

Communications connections 712 are devices, such as a wired networkinterface, or wireless network interface, which interface withcommunication media to transmit data over and receive data from signalpropagated over the communication media.

The computer 700 may have various input device(s) 714 such as a pointerdevice, keyboard, touch-based input device, pen, camera, microphone,sensors, such as accelerometers, thermometers, light sensors and thelike, and so on. The computer 700 may have various output device(s) 716such as a display, speakers, and so on. Such devices are well known inthe art and need not be discussed at length here.

The various computer storage devices 708 and 710, communicationconnections 712, output devices 716 and input devices 714 can beintegrated within a housing with the rest of the computer, or can beconnected through various input/output interface devices on thecomputer, in which case the reference numbers 708, 710, 712, 714 and 716can indicate either the interface for connection to a device or thedevice itself as the case may be. The various modules, tools, orapplications, and data structures and flowcharts of FIGS. 1-6, as wellas any operating system, file system and applications on a computer inFIG. 7, can be implemented using one or more processing units of one ormore computers with one or more computer programs processed by the oneor more processing units. A computer program includescomputer-executable instructions and/or computer-interpretedinstructions, such as program modules, which instructions are processedby one or more processing units in the computer. Generally, suchinstructions define routines, programs, objects, components, datastructures, and so on, that, when processed by a processing unit,instruct or configure the computer to perform operations on data, orconfigure the computer to implement various components, modules or datastructures.

In one aspect, an article of manufacture includes at least one computerstorage medium, and computer program instructions stored on the at leastone computer storage medium. The computer program instructions, whenprocessed by a processing system of a computer, the processing systemcomprising one or more processing units and storage, configures thecomputer as set forth in any of the foregoing aspects and/or performs aprocess as set forth in any of the foregoing aspects.

Any of the foregoing aspects may be embodied as a computer system, asany individual component of such a computer system, as a processperformed by such a computer system or any individual component of sucha computer system, or as an article of manufacture including computerstorage in which computer program instructions are stored and which,when processed by one or more computers, configure the one or morecomputers to provide such a computer system or any individual componentof such a computer system.

It should be understood that the subject matter defined in the appendedclaims is not necessarily limited to the specific implementationsdescribed above. The specific implementations described above aredisclosed as examples only.

What is claimed is:
 1. A computer system comprising: a processing systemcomprising computer storage and a processing device that executescomputer program instructions; an overlay module comprising computerprogram instructions that, when executed by the processing system,overlay a calibrated image from digital motion video on a rendered imageof a three-dimensional model of a scene according to an opacity on adisplay, wherein the digital motion video originates from a camera thathas a view of the scene, from a period of time, and wherein the imagerendered from the three-dimensional model of the scene is for a viewbased on a location of the camera in the scene; and wherein, in responseto an input related to locations in the overlaid calibrated image, theprocessing system provides the input to three-dimensional modelingsoftware providing the three-dimensional model.
 2. The computer systemof claim 1, further comprising: a calibration module comprising computerprogram instructions that, when executed by the processing system,generate the calibrated image from the digital motion video based oncorrespondence with the image rendered from a three-dimensional model.3. The computer system of claim 1, wherein the overlay control modulegenerates a graphical user interface on the display enabling a user toprovide input to modify the opacity.
 4. The computer system of claim 1,further comprising the three-dimensional modeling software, wherein thethree-dimensional modeling software comprises computer programinstructions that, when executed by the processing system, enables auser to manipulate the three-dimensional model while the calibratedimage is overlaid on the rendered image of the three-dimensional model.5. The computer system of claim 1, further comprising: an imageprocessing module comprising computer program instructions that, whenexecuted by the processing system, correct spatial distortions in thedigital motion video due to the camera, and wherein the calibrated imageis generated from corrected digital motion video.
 6. The computer systemof claim 1, further comprising: a playback module comprising computerprogram instructions that, when executed by the processing system, playsback calibrated images from the digital motion video, in response touser inputs; and wherein the overlay control module overlays the playedback calibrated images on the image rendered of the three-dimensionalmodel.
 7. The computer system of claim 2, wherein the calibration modulegenerates a two-dimensional transform for calibrating an image from thedigital motion video to the image rendered of the three-dimensionalmodel.
 8. A computer program product comprising: computer storage havingcomputer program instructions stored thereon, wherein the computerprogram instructions, when processed by a processing system comprising aprocessing device, configures the processing system to be comprising: anoverlay module comprising computer program instructions that, whenexecuted by the processing system, overlay a calibrated image fromdigital motion video on a rendered image of a three-dimensional model ofa scene according to an opacity on a display, wherein the digital motionvideo originates from a camera that has a view of the scene, from aperiod of time, and wherein the image rendered from thethree-dimensional model of the scene is for a view based on a locationof the camera in the scene; and wherein, in response to an input relatedto locations in the overlaid calibrated image, the processing systemprovides the input to three-dimensional modeling software providing thethree-dimensional model.
 9. The computer program product of claim 8,wherein the computer program instructions are further comprising: acalibration module comprising computer program instructions that, whenexecuted by the processing system, generate the calibrated image fromthe digital motion video based on correspondence with the image renderedfrom a three-dimensional model.
 10. The computer program product ofclaim 8, wherein the overlay control module causes the processing systemto generate a graphical user interface on the display enabling a user toprovide input to modify the opacity.
 11. The computer program product ofclaim 8, wherein the computer program instructions are furthercomprising the three-dimensional modeling software, wherein thethree-dimensional modeling software comprises computer programinstructions that, when executed by the processing system, enables auser to manipulate the three-dimensional model while the calibratedimage is overlaid on the rendered image of the three-dimensional model.12. The computer program product of claim 8, wherein the computerprogram instructions are further comprising: an image processing modulecomprising computer program instructions that, when executed by theprocessing system, correct spatial distortions in the digital motionvideo due to the camera, and wherein the calibrated image is generatedfrom corrected digital motion video.
 13. The computer program product ofclaim 8, wherein the computer program instructions are furthercomprising: a playback module comprising computer program instructionsthat, when executed by the processing system, plays back calibratedimages from the digital motion video, in response to user inputs; andwherein the overlay control module overlays the played back calibratedimages on the image rendered of the three-dimensional model.
 14. Thecomputer program product of claim 9, wherein the calibration modulegenerates a two-dimensional transform for calibrating an image from thedigital motion video to the image rendered of the three-dimensionalmodel.
 15. A computer-implemented process performed by a processingsystem comprising a processing device and computer storage havingcomputer program instructions stored thereon, wherein the computerprogram instructions, when processed by the processing device, performsthe process comprising: overlaying a calibrated image from digitalmotion video on a rendered image of a three-dimensional model of a sceneaccording to an opacity on a display, wherein the digital motion videooriginates from a camera that has a view of the scene, from a period oftime, and wherein the image rendered from the three-dimensional model ofthe scene is for a view based on a location of the camera in the scene;and in response to an input related to locations in the overlaidcalibrated image, providing the input to three-dimensional modelingsoftware providing the three-dimensional model.
 16. Thecomputer-implemented process of claim 15, further comprising: generatingthe calibrated image from the digital motion video based oncorrespondence with the image rendered from a three-dimensional model.17. The computer-implemented process of claim 15, further comprisinggenerating a graphical user interface on the display enabling a user toprovide input to modify the opacity.
 18. The computer-implementedprocess of claim 15, further comprising, in response to input from auser, manipulating the three-dimensional model while the calibratedimage is overlaid on the rendered image of the three-dimensional model.19. The computer-implemented process of claim 15, further comprising:correcting spatial distortions in the digital motion video due to thecamera, and wherein generating the calibrated image is performed usingthe corrected digital motion video.
 20. The computer-implemented processof claim 15, further comprising: playing back calibrated images from thedigital motion video in response to user inputs; wherein the played backcalibrated images are overlaid on the image rendered of thethree-dimensional model.